Mastering Apache Spark 2.x by Romeo Kienzler

Author:Romeo Kienzler
Language: eng
Format: mobi
Tags: COM062000 - COMPUTERS / Data Modeling and Design, COM018000 - COMPUTERS / Data Processing, COM051280 - COMPUTERS / Programming Languages / Java
Publisher: Packt
Published: 2018-07-18T10:35:15+00:00

object bayes1 extends App {

The HDFS data file is again defined, and a Spark context is created as before:

val hdfsServer = "hdfs://localhost:8020"

val hdfsPath = "/data/spark/nbayes/"

val dataFile = hdfsServer+hdfsPath+"DigitalBreathTestData2013-MALE2a.csv"

val sparkMaster = "spark://loclhost:7077"

val appName = "Naive Bayes 1"

val conf = new SparkConf()

conf.setMaster(sparkMaster)

conf.setAppName(appName)

val sparkCxt = new SparkContext(conf)

The raw CSV data is loaded and split by the separator characters. The first column becomes the label (Male/Female) that the data will be classified on. The final columns separated by spaces become the classification features:

val csvData = sparkCxt.textFile(dataFile)

val ArrayData = csvData.map {

csvLine =>

val colData = csvLine.split(',')

LabeledPoint(colData(0).toDouble,

Vectors.dense(colData(1)

.split('')

.map(_.toDouble)

)

)

}

The data is then randomly divided into training (70%) and testing (30%) datasets:

val divData = ArrayData.randomSplit(Array(0.7, 0.3), seed = 13L)

val trainDataSet = divData(0)

val testDataSet = divData(1)

The Naive Bayes MLlib function can now be trained using the previous training set. The trained Naive Bayes model, held in the nbTrained variable, can then be used to predict the Male/Female result labels against the testing data:

val nbTrained = NaiveBayes.train(trainDataSet)

val nbPredict = nbTrained.predict(testDataSet.map(_.features))

Given that all of the data already contained labels, the original and predicted labels for the test data can be compared. An accuracy figure can then be computed to determine how accurate the predictions were, by comparing the original labels with the prediction values:

val predictionAndLabel = nbPredict.zip(testDataSet.map(_.label))

val accuracy = 100.0 * predictionAndLabel.filter(x => x._1 == x._2).count() /

testDataSet.count()

println( "Accuracy : " + accuracy );

}

So, this explains the Scala Naive Bayes code example. It's now time to run the compiled bayes1 application using spark-submit and determine the classification accuracy. The parameters are the same. It's just the class name that has changed:

spark-submit \

--class bayes1 \

--master spark://hc2nn.semtech-solutions.co.nz:7077 \

--executor-memory 700M \

--total-executor-cores 100 \

/home/hadoop/spark/nbayes/target/scala-2.10/naive-bayes_2.10-1.0.jar

The resulting accuracy given by the Spark cluster is just 43 percent, which seems to imply that this data is not suitable for Naive Bayes:

Accuracy: 43.30

Luckily we'll introduce artificial neural networks later in the chapter, a more powerful classifier. In the next example, we will use K-Means to try to determine what clusters exist within the data. Remember, Naive Bayes needs the data classes to be linearly separable along the class boundaries. With K-Means, it will be possible to determine both: the membership and centroid location of the clusters within the data.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.